Upgrade to text-embedding-3-large model as default, with vector storage optimizations #2470

pamelafox · 2025-04-01T17:50:44Z

Purpose

This pull request changes the default embedding model to text-embedding-3-large, with 3072 dimensions, along with these AI Search vector storage optimizations:

Truncate dimensions to 1024
Binary quantization
Preserve originals in the search index for rescoring
Don't store the vectors in the search index itself

See this notebook for a demonstration of the effects of those optimizations. Due to the rescoring, the search quality remains high.

This PR introduces a new environment variable AZURE_SEARCH_FIELD_NAME_EMBEDDING so that developers can theoretically have multiple fields in their index, for different embedding sizes/models.

This PR also changes the SKU for all models to GlobalStandard. It's becoming really tricky to find a region for the Standard SKU that works for all the models. Some developers may not be comfortable with GlobalStandard, depending on their regulations, so they can still change the SKU manually as desired.

Fixes #2383

Does this introduce a breaking change?

When developers merge from main and run the server, azd up, or azd deploy, will this produce an error?
If you're not sure, try it out on an old environment.

[X] Yes - I am trying to make it backwards compatible, but it's hard! I suspect that developers that recently deployed gpt-4o-mini with Standard sku will get an error, and need to run `azd env set` to change the deployment name or sku name.
[ ] No

Does this require changes to learn.microsoft.com docs?

This repository is referenced by this tutorial
which includes deployment, settings and usage instructions. If text or screenshot need to change in the tutorial,
check the box below and notify the tutorial author. A Microsoft employee can do this for you if you're an external contributor.

[ ] Yes
[X] No

Type of change

[ ] Bugfix
[X] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

Code quality checklist

See CONTRIBUTING.md for more details.

The current tests all pass (python -m pytest).
I added tests that prove my fix is effective or that my feature works
I ran python -m pytest --cov to verify 100% coverage of added lines
I ran python -m mypy to check for type errors
I either used the pre-commit hooks or ran ruff and black manually on my code.

github-actions · 2025-04-02T23:08:31Z

Check Country Locale in URLs

We have automatically detected added country locale to URLs in your files.
Review and remove country-specific locale from URLs to resolve this issue.

Check the file paths and associated URLs inside them.
For more details, check our Contributing Guide.

File Full Path Issues

docs/deploy_features.md

#	Link	Line Number
1	`https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?tabs=global-standard%2Cstandard-chat-completions#models-by-deployment-type`	`197`

Copilot

Pull Request Overview

This PR upgrades the default embedding model to "text-embedding-3-large" (3072 dimensions) and implements several vector storage optimizations including truncation, binary quantization, and preserving original values for rescoring. It also introduces new environment variables for embedding field names and updates documentation and related code to support the new configuration.

Updated tests and environment variables for the new embedding model and dimensions.
Revised documentation to reflect model and deployment changes.
Refactored search management and approach modules to use dynamic embedding field names.

Reviewed Changes

Copilot reviewed 24 out of 27 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/conftest.py	Updated mocked model and dimensions for the new embedding model.
docs/gpt4v.md	Changed embedding model reference from ada to text-embedding-3-large.
docs/deploy_features.md	Updated deployment instructions and embedding model references.
docs/deploy_existing.md	Adjusted instructions for existing deployments to use the new model.
azure.yaml	Added new environment variables for embedding field names.
app/backend/prepdocslib/searchmanager.py	Refactored index creation to use dynamic embedding field names and profiles.
app/backend/integratedvectorizerstrategy.py	Passed new search field names into indexer skill configuration.
app/backend/prepdocs.py	Updated embedding field configuration from environment variables.
app/backend/approaches/*	Modified constructors and vector field usages to accept new embedding field.
app/backend/app.py	Integrated new environment variables for embedding field names in client setup.
.github/workflows/azure-dev.yml & .azdo/pipelines/azure-dev.yml	Included new environment variable exports for embedding field names.

Files not reviewed (3)

app/backend/requirements.txt: Language not supported
infra/main.bicep: Language not supported
infra/main.parameters.json: Language not supported

Copilot · 2025-04-02T23:38:44Z

docs/deploy_features.md

+By default, the deployed Azure web app uses the `text-embedding-3-large` embedding model. If you want to use a different embeddig model, you can do so by following these steps:



There's a typo in the word 'embeddig'. Please change it to 'embedding'.

Suggested change

By default, the deployed Azure web app uses the `text-embedding-3-large` embedding model. If you want to use a different embeddig model, you can do so by following these steps:

By default, the deployed Azure web app uses the `text-embedding-3-large` embedding model. If you want to use a different embedding model, you can do so by following these steps:

pamelafox · 2025-04-03T00:09:54Z

app/backend/approaches/approach.py

            "id": self.id,
            "content": self.content,
+            # Should we rename to its actual field name in the index?


This is a pending question as to whether to send it down using the field of the index?

pamelafox · 2025-04-03T00:11:42Z

app/backend/approaches/approach.py

@@ -317,7 +321,8 @@ class ExtraArgs(TypedDict, total=False):
            **dimensions_args,
        )
        query_vector = embedding.data[0].embedding
-        return VectorizedQuery(vector=query_vector, k_nearest_neighbors=50, fields="embedding")
+        # TODO: use optimizations from rag time journey 3


No changes are made to the actual search request, right?
We didn't make any here:
https://github.com/microsoft/rag-time/blob/main/Journey%203%20-%20Optimize%20your%20Vector%20Index%20for%20Scale/sample/3-Vector-Compression.ipynb

pamelafox · 2025-04-03T00:12:35Z

app/backend/approaches/chatreadretrievereadvision.py

-                    if field == "embedding"
-                    else await self.compute_image_embedding(query_text)
+                    await self.compute_image_embedding(query_text)
+                    if field.startswith("image")


This was a bit tricky, feels a bit code smelly.

pamelafox · 2025-04-03T00:12:48Z

app/backend/approaches/retrievethenreadvision.py

-                    if field == "embedding"
-                    else await self.compute_image_embedding(q)
+                    await self.compute_image_embedding(q)
+                    if field.startswith("image")


Same logic here

pamelafox · 2025-04-03T00:13:09Z

app/backend/approaches/retrievethenreadvision.py

@@ -83,7 +85,7 @@ async def run(
        minimum_reranker_score = overrides.get("minimum_reranker_score", 0.0)
        filter = self.build_filter(overrides, auth_claims)

-        vector_fields = overrides.get("vector_fields", ["embedding"])
+        vector_fields = overrides.get("vector_fields", [self.embedding_field])


Similar here, need to decide whether the frontend should render the actual field names.

pamelafox and others added 12 commits March 28, 2025 09:47

Initial changes for text-embedding-3

e4d98ac

Change text-embedding-3

2fddbd3

Bicep fixes

3e6d743

More embedding related changes

1b3d100

Merge branch 'main' into embedding3

7ab5bfe

Add dimension truncation

d2b2e9f

Mypy fix

6f07ce8

Fix mypy issues

121521a

Fix tests, add parameter

bfb74e6

Upgrade int vect for new embedding model

e9b822c

Merge branch 'main' into embedding3

430f522

Add missing env vars in other files

78cd4c1

Remove en-us from markdown

14713d2

pamelafox requested review from Copilot and mattgotteiner April 2, 2025 23:38

Copilot AI reviewed Apr 2, 2025

View reviewed changes

Merge branch 'main' into embedding3

6e76618

pamelafox marked this pull request as ready for review April 3, 2025 00:09

pamelafox commented Apr 3, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to text-embedding-3-large model as default, with vector storage optimizations #2470

Upgrade to text-embedding-3-large model as default, with vector storage optimizations #2470

pamelafox commented Apr 1, 2025 •

edited

Loading

github-actions bot commented Apr 2, 2025

Copilot AI left a comment

Copilot AI Apr 2, 2025

pamelafox Apr 3, 2025

pamelafox Apr 3, 2025

pamelafox Apr 3, 2025

pamelafox Apr 3, 2025

pamelafox Apr 3, 2025

		By default, the deployed Azure web app uses the `text-embedding-3-large` embedding model. If you want to use a different embeddig model, you can do so by following these steps:

Upgrade to text-embedding-3-large model as default, with vector storage optimizations #2470

Are you sure you want to change the base?

Upgrade to text-embedding-3-large model as default, with vector storage optimizations #2470

Conversation

pamelafox commented Apr 1, 2025 • edited Loading

Purpose

Does this introduce a breaking change?

Does this require changes to learn.microsoft.com docs?

Type of change

Code quality checklist

github-actions bot commented Apr 2, 2025

Check Country Locale in URLs

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Copilot AI Apr 2, 2025

Choose a reason for hiding this comment

pamelafox Apr 3, 2025

Choose a reason for hiding this comment

pamelafox Apr 3, 2025

Choose a reason for hiding this comment

pamelafox Apr 3, 2025

Choose a reason for hiding this comment

pamelafox Apr 3, 2025

Choose a reason for hiding this comment

pamelafox Apr 3, 2025

Choose a reason for hiding this comment

pamelafox commented Apr 1, 2025 •

edited

Loading